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Abstract. The Busemann-Petty problem asks whether origin symmet- 
ric convex bodies in R" with smaller hyperplane sections necessarily 
have smaller volume. The answer is affirmative if n < 3 and negative if 
n > 4. We consider a class of convex bodies that have a certain invari- 
ance property with respect to their ordered K-tuples of coordinates in 
R"" and prove the corresponding problem. 



1. Introduction 

In 1956 the Busemann-Petty problem was posed (see [BP]), asking the 
following question: 

Suppose that K and L are two origin symmetric convex bodies in M" such 
that for every ^ G S^~^, 

Voi„_i {K n e^) < Voi„_i (L n e^) . 

Does it follow that 

Vol„(i^) < Vol„(L) ? 

The answer is affirmative if n < 4 and negative if n > 5. The problem was 
solved in the late 90's as a result of a series of papers ( [Ba], [Bo], [Gal], 
[Ga2], [GKS], [Gi], [H], [Kl], [K2], [LR], [Lu], [Pa], [Zhl], [Zh2], ; see [K8, 
p. 3] for the history of the solution). 

In this article we consider the corresponding problem in M^'^ for convex 
bodies that have the property of invariance with respect to certain rotations 
considering the volumes of the nn — K-dimensional central sections, where 
At G N. 

Let X = {xi,X2, . . . , Xftra) G M*^". For every a G SO{k) we define 

Ra{x) := (cr(xi, . . . , X^), • • • , Cj(x^(„_i) + i, . . . , X^n)) 

to be the vector that corresponds to the rotation of the ordered K-tuples of 
coordinates of x by a. 

We consider the class of convex bodies that are invariant under any 
rotations: if x = {xi,X2, ■ ■ ■ , x^n) & K"^", and a G SO{k), then 

\\^\\d = {{{o'i^l, ■ ■ ■ , Xk), ■ ■ ■ , 0'{Xi^(^^_i^^i, . . . , X^n)) ||d- 

We then say that the body D has the Ru -invariance property or that it is 
Rfj-invariant. 
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Now suppose {/rf, (Ti, . . . , cJk-i} C SO{k) is an orthogonal sequence of 
counterclockwise rotations. Then the set {Id, Rai, ■ ■ ■ , Ra^-i} is also or- 
thogonal, namely, for every x G M'^"' the vectors x, R^j^ (x), . . . , Ra^-i (x) are 
mutually orthogonal. For ^ G S*""^^, we denote by Ti^ the K-dimensional 
subspacc of M''" generated by {^j(0}j=oi (where Rq = Id)- 

Now, the problem can be formulated as follows: 
Suppose K and L are two origin symmetric, with the i?cr-invariance property, 
convex bodies in R**", so that 

\o\^n-K{K n H^) < VoU_«(L n H^), 

for every 

^ g Does it follow that 

VoU„(i^) < VoU.„(L) ? 

Note that when k = 1 the problem corresponds to the real Busemann- 
Petty problem and for k = 2 the problem coincides with the complex 
Buscmann-Petty problem, which has affirmative answer if n < 3 and nega- 
tive if n > 4. 

In this article we prove that the answer to the Busemann-Petty problem 
for bodies with the i2o-invariance property is affirmative in the cases where 
(i) n = 2, KG N, (a) n = 3, k <2, (iii) n = 4 and k = 1 and negative for 
any other values of n and n. 

A related result on the Busemann-Petty problem for bodies that satisfy 
an invariance property can be found in [Ru]. 



2. The analytic connection to the problem 

Throughout this paper we use the Fourier transform of distributions. The 
Schwartz class of rapidly decreasing infinitely differentiable functions (test 
functions) in is denoted by 5(M"'), and the space of distributions over 
5(M") by 5'(M"). The Fourier transform atf of a distribution / G S'{W) is 
defined by (/, (p) = (/, cp) for every test function 4>. A distribution is called 
even homogeneous of degree p G R if {f{x),<p{x/a)) = \a\^'^P{f, 0) for every 
test function (j) and every a G M, a. ^ 0. The Fourier transform of an even 
homogeneous distribution of degree p is an even homogeneous distribution 
of degree —n — p. A distribution / is called positive definite if, for every test 
function cf), (/, * <p{—x)) > 0. By Schwartz's generalization of Bochner's 
theorem, this is equivalent to / being a positive distribution in the sense 
that > for every non-negative test function (p, (see [K8, section 2.5] 

for more details). 

We denote by A the Laplace operator and by | • I2 the Euclidean norm in 

the proper space. 

A compact set K C M" is called a star body, if every straight line that 
passes through the origin crosses the boundary of the set at exactly two 
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points and the boundary of K is continuous in the sense that the Minkowski 
functional of K, defined by 



is a continuous function on M". Using polar coordinates it is possible to 
obtain the following polar formula for the volume of the body: 



A star body K in M" is called fc-smooth (infinitely smooth) if the restric- 
tion of to the sphere 5"-^ belongs to the class of (C°°(5"-i)). 
It is well-known that one can approximate any convex body in in the 
radial metric, d{K,L) = sup{|/3x(0 ~ Pl{0\^ C ^ S^'-^^}, by a sequence of 
infinitely smooth convex bodies. The proof is based on a simple convolution 
argument (see for example [Sch, Theorem 3.3.1]). The same idea can be 
used to show that any convex body in W^"^ with the i?cr-invariance property 
can be approximated in the radial metric by a sequence of infinitely smooth 
convex bodies that are also i?o-invariant. This follows from the fact that 
the i?CT-invariance is preserved under convolutions. 

If D is an infinitely smooth origin symmetric star body in and < < 
n, then the Fourier transform of the distribution 11 homogeneous 
function of degree — n + A; on M", whose restriction to the sphere is infinitely 
smooth (see [K8, Lemma 3.16]). 

The following Proposition is a spherical version of Parseval's formula es- 
tablished in [K4], (see also [K8, Lemma 3.22]): 

Proposition 1. Let K and L he two infinitely smooth origin symmetric 
convex bodies in and < p < n. Then 



The class of intersection bodies was introduced by Lutwak [Lu]. This 
class was generalized in [K3], as follows: Let 1 < A; < n, and let D and L be 
two origin symmetric star bodies in M"". We say that D is the k-intersection 
body of L if for every (n — /c)— dimensional subspace H of 



More generally, we say that an origin symmetric star body D in is a 
k-intersection body if there exists a finite Borel measure on so that 
for every even test function ^ G <S(M"), 



Note that ^-intersection bodies of star bodies are those ^-intersection bodies 
for which the measure has a continuous strictly positive density; see [K7] 



x\\k = minjo! > : x G aK}, 




(LnH). 




or [K8, p. 77]. 
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A more general concept of embedding in L_p was introduced in [K6]. 
Let D be an origin symmetric star body in M", and X = (R", || • \\d)- For 
< p < n, we say that X embeds in L_p if there exists a finite Borel measure 
ji on S'"'~^ so that, for every even test function ^ 

/ ||a;||~^.^(a:) dx = ! ( f \z\P-^4>{ze) dz^ dn{e). 

Obviously, an origin symmetric star body D in is a fc-intersection body 
if and only if the space (M", || • \\d) embeds in -L_fc. In this article we use 
embeddings in L_p only to state some results in continuous form; for more 
applications of this concept, see [K8, Ch. 6]. 

Embeddings in L_p and /c-intersection bodies admit a Fourier analytic 
characterization that we are going to use throughout this article: 

Proposition 2. ([K7], [K8, Th. 6.16]) Let D he an origin symmetric star 
body in M", < p < n. The space (M", || • embeds in L^p if and only 
if the function represents a positive definite distribution on M". In 

particular, D is a k -intersection body if and only if HxH^'^ is a positive 
definite distribution on R". 

Another important fact that will be used is the co-called second derivative 
test, (see [K8, Theorems 4.19, 4.21]). 

Proposition 3. Let A > 3, A; G N U {0}, q > 2 and let Y be a finite 

dimensional normed space of dimension greater or equal to A. Then the q- 
sum of M and Y does not embed in L^p with < p < X — 2. In particular, 
this direct sum is not a k-intersection body for any 1 < k < X — 2. 

Recall that the g-sum of M with a space Y, Y (Bq M, is defined as the space 
of pairs {{t,y) : eR,y €Y} with the norm \\{y,t)\\ = {\\y\\^ + 

Let K > 1 and let H be an (ku — K)-dimensional subspacc of R"". Fix any 
orthonormal basis ei,...,eK in the orthogonal subspace H-^. For a convex 
body D in M''", define the {nn — k) -dimensional parallel section function 
Ad,h as a function on M*^ such that 

Ad,h{u) = Vol {Dn{H + uiei + ... + u^e^}) 

xi\\x\\D)dx, ueR\ (1) 

{x&R'^"-:{x,ei)=ui,...,{x,ek)=Ui^} 

If D is infinitely smooth the function Ajj^h is infinitely differentiable at 

the origin (sec [K8, Lemma 2.4]). So we can consider the action of the 
distribution |u|^'^~''/r(— (7/2) on Ad^h and apply a standard regularization 
argument (see for example [KB, p. 36] and [GS, p. 10]). Then the function 

, I \~q—K 

/ \u\ \ 
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is an entire function of g € C. If g = 2m, m G N U {0}, then 

-q—K 

,Ad,h{u)] 



r(-g/2) 



q=2m 



where |S'''~^| = 27r'*/^/r(K/2) is the surface area of the unit sphere S'^~^ in 
M**. When the body D is origin symmetric, the function Ad^h is even, and 
for < g < 2 we have (see also [K8 p. 39]) 

I —q—K 

-,Ad,h{u)' 



-9/2) 

T{-q/2) Js.n-i \Jo J ^ ' 

Note that the function (2) is equal (up to a constant) to the fractional power 
of the Laplacian A5/^A£)^ii-,(see [KKZ] or [K7] for complete definition). 

The following proposition was proved in [KKZ, Prop. 4]. 

Proposition 4. Let D he an infinitely smooth origin symmetric convex body 
in M**" and \ < k < nn. Then for every (nn — k)- dimensional subspace H 
ofW^'^ and any g G M, —k < q < nn — k, 

,Ad,h{u) 



[ {Md^"^''^")''(.G) de. (5) 



r{{q + K)/2){Kn-q-K) 
Also for every m G N U {0}, m < {nn — k)/2, 

(_-\\m r 

A^AnMO) = \ V / (IkllB""^'"^^")^^) dv- (6) 

[lirp^nn — Am — n) J gKn-i^fj± ' 

Brunn's theorem (see for example [K8, Theorem 2.3]) states that for an 
origin symmetric convex body and a fixed direction, the central hyperplane 
section has the maximal volume among all the hyperplane sections per- 
pendicular to the given direction. As a consequence we have the following 
generalization proved in [KKZ, Lemma 1]. 

Lemma 1. If D is a 2-smooth origin symmetric convex body in M"", then 
the function Ad,h is twice differentiable at the origin and 

AAd,h{^) < 0- 
Besides that for any q € (0, 2), 

''''''' ^-,Ad,h{^))>0- 



r(-g/2) ■ 
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3. Connection with k-intersection bodies 

Now, we are ready to connect the i?o--invariance property of the bod- 
ies with the Fourier trasform of their norm that wih yield the solution to 
the problem. The following simple observation is crucial for applying the 
analytic methods to convex bodies that carry the i^^-invariance property. 

Lemma 2. Suppose that D is an origin symmetric infinitely smooth, with 
the Ra-invariance property, star body in M"". Then for every < a < ku 
and ^ G S'^^"^ the Fourier transform of the distribution ||ic||^" is a constant 
function on S"'""^ flH^. 

Proof : By [K8, Lemma 3.16], since D is infinitely smooth, (||a:^||^")^ is a 
continuous function outside of the origin in M'*". In addition, since the body 
D has the i?o--invariance property, by the connection between the Fourier 
transform of distributions and linear transformations, the Fourier transform 
of is also i?o--invariant. Now, as mentioned in the Introduction, the 

K-dimensional subspace of M*^" is spanned by ^, • • • jRcr^-iiO- 
Hence, every vector in the (k — l)-dimcnsional sphere S'^"'~^ fl Ti^ is the 
image of ^ under one of the coordinate- wise rotations R^, so the Fourier 
transform of is a constant function on S'^^~^ fl 7^^. □ 

We use the above lemma to express the volume of K-codimensional sec- 
tions in terms of the Fourier transform. 

Theorem 1. Let K be an infinitely smooth origin symmetric, with the R^- 
invariance property, convex body in W^'^,n > 2. For every ^ G S'^^"^, we 
have 

Proof : Let ^ G S"'"-^ In formula (6) we put H = and m = 0. Then, 
by the definition of the (/en — /c)-dimensional section function (equation (1)), 
we have that 

Ao.n,(0) = Voi.„_.(i.n«,) = p^p^ {M-,'"-r (X)* 

By Lemma 2, the function under the integral is constant on the (k — 1)- 
dimensional sphere 5**"-^ fl Tif. Since ^ G S"*""^ nnf,we get that 

VoU_.(i.n«,) = (Ikt—)" (0. 

which proves the theorem, using the volume of the unit sphere of R*^. □ 

As in the case of the real Busemann-Petty (for k = 1) and the com- 
plex Busemann-Petty (where k = 2), the solution to this problem is closely 
related to the ^-intersection bodies. 
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Theorem 2. The answer to the Busemann- Petty problem for convex bodies 
with the Ra-invariance property is affirmative if and only if every origin 
symmetric -invariant convex body in R**" is a K-intersection body. 

The proof of the theorem follows by the next two lemmas. Note that we 
may assume that the bodies are infinitely smooth. This follows from the 
fact that one can approximate the two bodies, in the radial metric, with 
infinitely smooth convex bodies that have the i^^-invariance property (see 
Section 2). Then, if the answer to the problem is affirmative for infinitely 
smooth bodies it is also true for the original bodies. 



Lemma 3. Suppose K and L are infinitely smooth origin symmetric, with 
the Ra-invariance property, convex bodies mM^" so that K is a K-intersection 
body. Then, if for every ^ G 5"^"~^ 

Vol„{K n n^) < Vol„{L n W^), (7) 

then 

Vokn{K) < Vokn{L). 

Proof : The bodies K and L arc infinitely smooth, so by [K8, Th. 3.16], 
the Fourier transform of the distributions || • ||^'*"^'', || • H^*^ and || • H^''"'^'* 
are continuous functions outside of the origin of M''". By Theorem 1, the 
inequality (7) becomes 

Also, since K is a K-intersection body. Proposition 2 implies that the Fourier 
transform of the distribution || • H^^" is a non- negative function. So, we can 
multiply both sides of the latter inequality by (|| • ||^'*)^ . We then integrate 
over and use Parseval's spherical version, Proposition 1, to have that 

{\\ ■ \\-k )\0 {M~r'-'')\Od^ 

l—l 

< I {\\-\\-K)\omi^''^''T{m 



which gives 



\x\\K''''dx< / WxW^" 11x111"''+'' dx. 



We use the polar formula for the volume and Holder's inequality, then the 
latter inequality becomes 



1 p n-l 



KnVoU(i^) < ( / \\x\\-^''''dx) " ( / \\x\\l''''dx) " , 
which proves the Lemma. □ 
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For the negative part we need a perturbation argument to construct a 
body that will give a counter-example to the problem. The new body im- 
mediately inherits the additional property of invariance with respect to the 
rotations of the original convex body. 

Lemma 4. Suppose that there exists an origin symmetric convex body L in 
R*^", with the R„-invariance property, which is not a n-intersection body. 
Then there exists an origin symmetric -invariant convex body K in R**" 
such that for every ^ € S"*"""*^, 

Vol„{K(^H^) < VoU-niLnn^), 

but 

Proof : We assume that the body L is infinitely smooth with positive 
curvature. By [K8, Lemma 3.16] the Fourier transform of the distribution 
is a continuous function on the unit sphere S''*""^. Moreover there 
exists an open subset Q of 5"^""^ in which < 0. Since L is invari- 

ant with respect to all rotations we may assume that has also this 
invariance property. 

We use a standard perturbation procedure for convex bodies, see for ex- 
ample [K8, p. 96] (similar argument was used in [KKZ, Lemma 5]): Consider 
a non-negative infinitely diflFerentiable even function g supported on Q that 
is also i?o-invariant. We extend it to a homogeneous function of degree —k 
on R''"'. By [K8, Lemma 3.16] its Fourier transform is an even homogeneous 
function of degree — kti+k on R''", whose restriction to the sphere is infinitely 

smooth: (5(a;/|x|2)|a;|2")^(j/) = Hy/\y\2)\y\2^''+'' , where h G 
We define a body K so that 

for small enough e > so that the body K is strictly convex. Note that 
K also has the i?(j-invariance property. We apply the Fourier transform to 
both sides and the latter becomes 

mrr^'-fiO = {\\x\\l''-+^)\0 - i^^regiO (8) 
since g is non-negative. By Theorem 1, this implies that for every ^ G gKu-i 

Voi„(Kni/^) < Vol 

Kn—K, 

Now, we multilpy both sides of (8) by (|| • H"**)^ and integrate over the 
S^^n-i Then 
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-i27:re [ {\\x\\l^)\09i0 

JSi^ri-l 

JSK.n-1 

since (||a;||Z'')^ < on the support of g. We apply Proposition 1 and the 
polar formula for the volume to obtain 

/ Ikllx "+1I^IIZ"^^ > «nVoW(L). 

As in Lemma 3, wc use Holder's inequality and the polar formula for the 
volume of K to obtain the inverse inequality for the /tn-dimensional volumes 
of the bodies. □ 

4. The solution 

In order to prove the main result we need to determine the dimensions 
and the values of k for which every origin symmetric, with the i?(^-invariance 
property, convex body in M*^" is a /^-intersection body. It is known (see [K5] 
or [K8, Corollary 4.9] with Proposition 2) that for every origin symmetric 
convex body D in M*^", n > 3, the space (M''"', || • Wu) embeds in L_p for 
p € [kn — 3, kn). In other words, every origin symmetric convex body D in 
R*^", is a {Kn — 3), {nn — 2), {nn — l)-intersection body. 

Remark: Let n > 3. We consider the unit ball 

B^^" = {xe R'^" : \\x\\q = {{xj + ■ ■ ■ + + ... 

••• + (<„-iHi + --- + -L)^)'/^<i}, 

with q > 2. The space (M'^", || • ||sjn) contains the g-sum of R with the 
{nn — K)-dimensional subspace, Y = (R'^""'^, || • ||g,tn-,t). Then by the sec- 
ond derivative test, Proposition 3, for X = Kn — n, the space Rffi^y does not 
embed in L_p, if p < Kn — k — 2. Moreover, it follows by a result of E.Milman 
[Mi] that the larger space cannot embed in for p < Kn — k — 2. Hence, if 
K > , the unit ball " provides a counter-example of an origin symmet- 
ric, with the i?o-invariance property, convex body that is not a K-intersection 
body. 

It remains to see what happens in the range [Kn — k — 2, Kn — 3). 

Theorem 3. Every origin symmetric, with the R^j-invariance property, con- 
vex body K in R"" is a {Kn — k — 2) -intersection body in the following cases: 
{i) n = 2,K> 3, {a) n = 3, K > 2, (m) n > 4, «; G N. 

Moreover, the space (R''", || • \\k) embeds in L-p for p G [Kn — k — 2, Kn), 
when {i) n = 2, K > 2, {ii) n > 3, k G N. 

Proof : We use Proposition 4 with m = 1 and H = 7i^. The condition 
1 < "^"^"^ . is valid for the three cases of the first part of the theorem. Hence, 
by (6) and Lemma 2, we have that 
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By the generalization of Brunn's theorem (Lemma 1), we conclude that 

for every ^ G 5^^""^. Hence, by Proposition 2, if is a (ren — k — 2)-intersection 
body for the above cases. 

Now let n = 2, k > 2, or n > 3 with k G N. By (5) with H = and 
Lemma 2, we have that for every q G (0, 2) 

Then Lemma 1 implies that for < g < 2, (Ha^H]^'^"^''''''*)^ > 0. The range 
of q and Proposition 2 indicate that the space (M**", || • embeds in L_p 

for p G [K,n — K — 2, nn). □ 

Using the above theorem we can now prove the main result of this article. 

Theorem 4. The Busemann-Petty problem for origin symmetric convex 
bodies that have the R^-invariance property in M*^" has affirmative answer 
only if: {i) n = 2, k E N, {ii) n = 3, k < 2, [Hi) n = A and k= 1. 

Proof : 

When n = 2 and k G N the answer is affirmative by Theorems 2, 3 and 
Proposition 2, since k = 2k — k > 2k — k — 2. 

If n = 3 and k < 2 then k > Sk — k — 2 and using Theorems 2, 3 and 
Proposition 2 we have that the answer is also affirmative. On the other 
hand, if k > 3 then k < 3k — k — 2. By the Theorem 2 and the Remark the 
latter implies that the answer to the problem is negative and the body B^'^ 
gives a counter-example of a body that is not ^-intersection body in M^**. 

For n = 4 and k = 1 then 1 = kti — k — 2. So the answer is also affirmative. 

Lastly, if n = 4 with fc > 2 or n > 5 and /« G N then ku — k — 2 > k. The 
same reasoning as before, applying Theorem 2 and the Remark, gives that 
the answer to the Busemann-Petty problem for origin symmetric convex 
bodies that have the i?o--invariance property is negative and provides a 
counter-example. 

□ 
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